Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support resolving environment variable references in matrix config #7682

Merged
merged 4 commits into from
Feb 29, 2024

Conversation

benbp
Copy link
Member

@benbp benbp commented Feb 14, 2024

sorry for all the formatting updates, vscode got more opinionated this time and I don't want to fight it

This adds the ability to reference environment variables to be resolved at runtime for matrix values, for example:

"matrix": {
  "foo": "env:bar"
}

The primary use case is so we can reference common values like LinuxPool that we define as top level variables in an azure pipelines context, so we don't need to repetitively hardcode them in all the matrix configs like we do today.

@benbp benbp added the Central-EngSys This issue is owned by the Engineering System team. label Feb 14, 2024
@benbp benbp self-assigned this Feb 14, 2024
@benbp benbp requested a review from a team as a code owner February 14, 2024 01:43
@benbp benbp requested a review from scbedd February 14, 2024 01:43
@benbp benbp force-pushed the benbp/matrix-env-var branch from cbe6d8c to b94e9e9 Compare February 14, 2024 01:46
$importPath = $_.Value
}
}
if ((!$matrix -and !$importPath) -or !$importPath) {
return $matrix, @(), @{}
return $matrix, @(), $displayNamesLookup
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also fixed a separate bug here where display names lookup was getting lost if we didn't have imports

Copy link
Member

@scbedd scbedd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You have a test-branch where you're actually trying this, but this code update looks fine to me. Nothing is jumping out at me 😆

@benbp
Copy link
Member Author

benbp commented Feb 14, 2024

@mikeharder for example, this is what I'm thinking of switching our matrix configs to:

"matrix: {
  "Agent": {
    "ubuntu": {
      "Pool": "env:LinuxPool",
      "Demands": "env:LinuxImageDemands"
      "osName": "env:LinuxOS"
    }
  }
}

Then we can define all these values in a single place like image.yml referenced by eng/pipelines/templates/variables/globals.yml

@benbp benbp force-pushed the benbp/matrix-env-var branch 2 times, most recently from 70aabf9 to 52e3eda Compare February 14, 2024 03:15
@Azure Azure deleted a comment from azure-sdk Feb 14, 2024
@azure-sdk
Copy link
Collaborator

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

@Azure Azure deleted a comment from azure-sdk Feb 14, 2024
@azure-sdk
Copy link
Collaborator

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

Comment on lines 386 to 387
$envKey = $flattened.Value.Replace("env:", "")
$value = [System.Environment]::GetEnvironmentVariable($envKey) ?? ""
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
$envKey = $flattened.Value.Replace("env:", "")
$value = [System.Environment]::GetEnvironmentVariable($envKey) ?? ""
$value = (Get-Item $flattened.Value -ErrorAction SilentlyContinue)?.Value

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To me the code is a bit less clear if you don't know how get-item works, and there could be issues in the future if the outer conditional gets removed, allowing file lookups, etc.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was just trying to avoid us needing to parse variables when PS already does that. In this case it is pretty simple so I'm not too worried about splitting and just getting the env variable directly. I'll leave it to you :)

Copy link
Member

@weshaggard weshaggard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

couple suggestions but looks good otherwise.

@azure-sdk
Copy link
Collaborator

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

@benbp benbp force-pushed the benbp/matrix-env-var branch from c46c2cf to 5b9cb4a Compare February 14, 2024 20:54
@azure-sdk
Copy link
Collaborator

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

@weshaggard
Copy link
Member

Closing this for now as we don't need this feature currently.

@weshaggard weshaggard closed this Feb 14, 2024
@benbp benbp reopened this Feb 15, 2024
@azure-sdk
Copy link
Collaborator

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

@azure-sdk
Copy link
Collaborator

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

@azure-sdk
Copy link
Collaborator

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

@benbp benbp enabled auto-merge (squash) February 29, 2024 21:22
azure-sdk added a commit to Azure/azure-sdk-for-js that referenced this pull request Feb 29, 2024
Sync eng/common directory with azure-sdk-tools for PR
Azure/azure-sdk-tools#7682 See [eng/common
workflow](https://github.com/Azure/azure-sdk-tools/blob/main/eng/common/README.md#workflow)

---------

Co-authored-by: Ben Broderick Phillips <[email protected]>
@benbp benbp merged commit 3ecd831 into Azure:main Feb 29, 2024
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Central-EngSys This issue is owned by the Engineering System team.
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

4 participants